Crowdsourcing Music Similarity Judgments using Mechanical Turk
نویسنده
چکیده
Collecting human judgments for music similarity evaluation has always been a difficult and time consuming task. This paper explores the viability of Amazon Mechanical Turk (MTurk) for collecting human judgments for audio music similarity evaluation tasks. We compared the similarity judgments collected from Evalutron6000 (E6K) and MTurk using the Music Information Retrieval Evaluation eXchange 2009 Audio Music Similarity and Retrieval task dataset. Our data show that the results are highly comparable, and MTurk may be a useful method for collecting subjective ground truth data. Furthermore, there are several benefits to using MTurk over the traditional E6K infrastructure. We conclude that using MTurk is a practical alternative of music similarity when it is used with some precautions.
منابع مشابه
Discovering User Perceptions of Semantic Similarity in Near-duplicate Multimedia Files
We address the problem of discovering new notions of userperceived similarity between near-duplicate multimedia files. We focus on file-sharing, since in this setting, users have a well-developed understanding of the available content, but what constitutes a near-duplicate is nonetheless nontrivial. We elicited judgments of semantic similarity by implementing triadic elicitation as a crowdsourc...
متن کاملUsing Crowdsourcing to Compare Document Recommendation Strategies for Conversations
This paper explores a crowdsourcing approach to the evaluation of a document recommender system intended for use in meetings. The system uses words from the conversation to perform just-in-time document retrieval. We compare several versions of the system, including the use of keywords, retrieval using semantic similarity, and the possibility for user initiative. The system’s results are submit...
متن کاملEvaluating Crowdsourcing through Amazon Mechanical Turk as a Technique for Conducting Music Perception Experiments
Online crowdsourcing marketplaces, such as the Amazon Mechanical Turk, provide an environment for cost-effective crowdsourcing on a massive scale, leveraging human intelligence, expertise, and judgment. While the Mechanical Turk is typically used by businesses to clean data, categorize items, and moderate content, the scientific community, too, has begun experimenting with it to conduct academi...
متن کاملNortheastern University Runs at the TREC13 Crowdsourcing Track
The goal of the TREC 2012 Crowdsourcing Track was to evaluate approaches to crowdsourcing high quality relevance judgments for images and text documents. This paper describes our submission to the Text Relevance Assessing Task. We explored three different approaches for obtaining relevance judgments. Our first two approaches are based on collecting a limited number of preference judgments from ...
متن کاملThe MediaEval 2013 Brave New Task: Emotion in Music
Music is composed to be emotionally expressive. Emotional associations of music thus provide an especially natural feature for music indexing and recommendation. Emotion in Music Task is a brave new task addressing emotional characterization of music. In addressing the difficulties of emotion annotation we have turned to crowdsourcing, using Amazon Mechanical Turk. The dataset consists entirely...
متن کامل